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ABSTRACT 

This report of the evaluation of th^ Office of 
Instructional Resources, Teaching Center at the OniVersity of Florida 
de.scribes an approach to the problem of ass€ssing/an instructional 
program developed for students «ho do not meet the admission 
Requirements of the state university system. program evaluation 
design not only provides a* model to study the ^voluticn of the 
program but also describes the applic^ion of/fcrmative evaluation to 
program development.. Since the conduct af thp entire evalua'tion is 
influenced by dissemination. reqijireUents, ccfmmunication problems an^ 
•their resolution are also tteated. Wie evalfuation addresses four 
areas-, of the. teaching center's activities/ the management 'system, 
tutor effectiveness; test-item quality, and client' satisfaction. 
(Author/HSE) / 
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• • . . . INTRODUCTION - ' . - , 

» One of the greatest educational experiments to grow* out of the ferment 

'- ^ . . ' > . - • ' ■ ' 

of the 1960 's ^sbeert the 'opening of the college and universxi:y admissions to 
** " ' * . ' 

all thcfse who .seek post-secondary education. ' M^y of those, who have pursued ^ 
op^rtunities in higher education have been ,di>f f ereht from ^he traditional . 
college-bound student, different in eduteitibnal preparedness, in socio-economic 
strata ar\d in attitudes towards themselves as learners. 

This report: o£' the evaluation of the Office of Ins tructionalj Resources 

Teaching Center describes an- approach to de^l with the problems of assessing 

* * • - • / . » ' » 

• *^ ' - . 

an instructional program developed in 1972 for students who did not meet the 

admission requirements of'jthe Stat^ Univ^jsslty System.^ Thd^ evaluation .design 
, n'ot only provides «a model •to stjidy the evolution of a program, but it also de- 
scriljes the 'application of formative evaluatipn to program development. The 
conduct of the entire evaluation is influenced by the dissemination requirements 
Therefore, this paper also addresses problems in communication and provide j con- 
crete »^examples of their resolution. ' ' * • , 

ESSENTIAL COMPONENTS 0F THE PROGRAM ' 
Th^ Teaching Center is an academic, instructional unit serying the st^u- 
^de'nts and faculty of the special course sections. Recifuiting of these students 
is^done by the Office of Admissions; orientation to tl)e Universj\v is conducted 
by the Special Services Pi;ogram; financial aid is made available' through the 
Financial jjlld ^Office and the Special Services Program. Support services such 
as health care, academic, career and personnel cox^nseling and career placement 
are made available through regular university agencies. " \ 



Thes^specialized components ;of the Teaching Center include: * , / 

Myiltiple Achieveinent Testing and Feedback / Instructors divide course- 
^ content into units ank write a bank o^ test questions in multiple-choice or 
short-answer forraat ror each unit which are* stored in the computer: A 
studfeat, when ^he feells he is ready, cames to the Teaching Center for testing, 
is administered a computer-generated test and has it scored oy a laboratory 
a3sistant imrnediatel>\ If he has met'the criterion- set by the instructo^, 

r ' - ' ' / ' ' ' 

' ' he has completed that; unit, ff he fails, he is given additional help 'and 

attempts the tpst a^eiin, responding to different items from.the item bank 

stored in the computer. In^addition, the Registrar has established an '^H"' 

■ grade which allows a student an additional three weeks at the clo^e of the 
\ K j ' • _ * ' 

quarter to comple?te work without ^penalty*^ - ' ^ \ " ' 

. ' ^ ' ' . ' 

Tutoring . A tutor is assigned tq each course taught through the Center. 

He works closely with the instructor, attends clas^ and as responsible for . 

contact4.ng and tutoring every stud'ent in' the course: .Tutoring is done <)n an^ 

individual and group basis, both befdre and after testi^gj depending upon 

the preference and ne^ds of -the students. - , ' . V,--^ 

Score Reporting . Each day test results are sfcpiT/ed.in the computer. ' 

From this data, weekly^ progress reports are generated for each student and 

each course section, and are made available ^0 instructors, tutors, aad the. 

director of the-Center.' Thus it is. possible at alJL times to monitor the 

progress 'of any individual student or the progress of an entire class oh any 

particular unit of v7ork; • ° \ ; 



During the two years a ^tudent may enroll-^ in .special sections h^ also 
enrol"* ' ^ - - - j= ^i^^ rt^ 



)lis, in regular cours^es of the .University. ^ As his performance ^cores 



on tests increase, .be is encbufaged to enroll in more non-Teac^vi^g* Center 
courses so*that*Jiis transiticyn. to the .academi,c maiostream is successful. 



In addition, instructors are encouraged to use, essay-type examinations and 
vrriting assignm.ents as part of their instructional activities. ' T-Zritin^ 
instructors at the Teaching Center work with students on -these assignments, 
SO they may develop academic XTifiting skills that are essential in upper 
division work* 



' FORMATIVE EVALUATION 

The formative evaluation^ encompasseffl four areas of the Center's 
activities* The four areas identified for study wer^ the management system, 
tutor effectiveness, item quality, and client satisfaction* :^-Tlie entire 
population of, students using Teaching Center services during "the fall 
-quarter, 1976 has b^n included in this analysis. 

Insults 



1) Management system: testing times . It was clear that some students^ ^^^^ 
in most cour^s^s took an i'Aordinat'ely long period of time to pompl^^^" '^tlel^'^<^^^'^' 



tests* The distributions , of testing times were generally positively skewed 

which indicated that the majority of J the testdng times were less th'an the 

average testing time with § few extr^ely long times. ; . 

Mktiagement systfem: grading times * Wide variability in grading times 

characterized most courses, 

I • * - \' ' • - 

Grading time should be at a minimum with little variability* This goal 

was intended to reduce th^ pressure on graders and to increase the- inter- 
action between stiideats and tutors* The data doe^ not indicate that the ^ 
vgoal was successfully achieved. ^ ^ , * ' ' 

Management system: ^ student flow . . On the ba^is of the inftirination 
gathered, it was clear that Mondays and Fridays were^ the Heaviest- testing ^ ^ 



flays.. The heaviest weeks were the fourth and elev-entK weeks . The busiast 
^ time periods were '10:30 a.m. and 1:30 p. n. 

/ Management system: testing deadlines . Charts were prepared to document 

the pattern of test taking behavior affected by the establishment of different 
deadlines for unit tests. The question was ooncerned j^ith whether or not a 
greater- number. of days in which a student^w^s allowed to 'take a unit test , 
would affect the flow of traffit: in thevCenter. It was found that testing 
longer than three day periods resulted in the same pattern; the students 
tended to wait until 'the last day regardless ^pf how many days were initially 



all'c?Wed (see Fig. 1).' 

• \ 



Fig. 1. -Testi*tig Deadlines 
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^ Managenvefrt system: repeat. testing on the Svime day . Another problem 

\ . " • • ' . ■ • . I / - • ■ - ' • 

in directing * the student flow :^n the Center ^appeare^ to be -related .to the. 
practice of repeating the sarrie test on the same dAy. ,An analysis of "the 
scores for those students wKo did repeat tests .on the'^same day found that 

. - * - ' • • 7 

fifty-five percent of the time the practice did not benefit the student 

. / . '* ^ ' ^ 

(see Table !)♦. 



Improved Score 





TABLE 1 . REPEAT TEEING ON THE SAME DAY 

FALL 1976 



Nonimproved Scores 



Total Impreved- T^^': 
'N - 201; k = 45 



Fail T^^ice 
N = 110; 7o =-25 

Pa$s - Fail 

'N =: ^7; % = 6 
/ 

Pass * Dfeciine 
N, ^ 48;- % = 11 

Pass -'^No Ofeange -. 
N = 60; 7o ^.^3 

_ ^ J 

Total Nonimproved 

N = ^5; % = 55 - 




Management system: repeat testing over^iin^;^ This question was con^ 

cerned with the optimum number of a^^t^n^T^s^^t^al 1 o w^foV^n;^ given unit test* 

The Cei^iter policy was to allow repeat testing. However, i^xwas felt that "a 

maxirpum number of attempts should be** established based upon infiqrmatioa 

which would indicate that further ^attempts were not benefici,al tb "fchW 

student and were conti^ibuting to congestion in the Center. Data from the g 

*fall quarter 1976 testing sessions were analyzed, aQd<a matrix of average 

percent corrects scores was 'prepared for "students i^7ho ratested the same 
t 

unit test (see TabL^2) 
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* * f • 

TABLE *2. MATRIX OF SCORES FOR STUDENTS Tv"iIO 
•. >^ • . ■ REPEAT THE SAl-IE TEST: FALL 1976 



' Number of 




Average 


Percent 


Correct . 




; Number of 


'Times Tested 




2 


3 ' 


4- 


5 


Students 


* / 1 


62.0 


A 


* 






• ,'570 


If 


57.8 




• > 






- • 716 


3. 

V 




57.8 


67.1 


\ 


'v- 

» 


. -333 . 




43.9 


51.4 


55.1 


55.8 • 




44 • 


♦ 

. . 5 


. 43. O" 


53.2 


55.6 


■ 47.1 *. 


59.5 


^ ' -10 


6 ' ■ 

■ # — 


47.5 


40.0 


70.0 


35. o" 


70.0 


# 

1 



The matrix revealed that students who took the same test twice increased 
their scores from 57»8 to 68.4 percent correct. Students who took a test three 
times increased their scores from 51.1 to 67 •! percent cor^^ct. ^ Students who 

'retested more than three times failed to improve beyond the third attempt. 
While students who took a test f)nly once received the highest initial scores,^ ' 

•those students wha retested twice or three times received the highest final. 



scores. • . y 



' Tutor effectiveness . . The data from the tutor activity rfecQr^ 

revealed' that graat variability existed in the number of tutoring sessions 
wittiin subject area^. as well as a'cross subject ^reas* The number of sessions ' 
appe^ed to be related more to particular tutors rather tl;ian differing course 
requirements. ' ^ — . ^ - 

While more tutoring occurred 'in the physical science and mathematics 
areas than in the humanities and . social ^science fields, it was not established 
that these differences were subject related. u 

An additional critfeVion for judging th^ success -of the tutoring program 
wa? ttie aqademiqll success of thd studentis who were tutored. The following 



tSible lists the distribution^ of grades in the courses faught' through the 
-^enteT~tn~t\reTtall.±91£^^^^ number of tutoring sessions for -ea<sh 

gra'de revealed a positive relationship isetwe^en the number of sessions and 

• \ • ' ' 

^ the grade received ft)r students* who were tested .(see ^lable 3). 



TABLfi^3. DISTRIBJJTION OF GRADES BY NUMBER 
^ • ^ , OF TUTORING SESSIONS 



Grade 


XTotal 
Number 


Total Number 
Tutored 


Total Number 
Sessions 


Average NuHiber 
Sessions 


.*- A 


69 


.36 , . 


216 




. B 


* 

99 ♦ 


60 


; ; 34b ' 


• 5.8^ ■ 


. C 


115 


74 • 


. 320 


• 4.3 


, • D 


■ 35 


'*29* . 


109 


I. 3-7 


E 




16 


40 * 


.2.5. 



Sessions 



1-30 
1-24 
1 r!l8 
1 - 12 ^ 
1 - ^8 



^One student received a "D"^ with 30 tutoring sessions and was deleted, 
from the analysis. 



3)# irem analysis / Item analyses were run for all unit tests in the 
physical science courses.' Item difficuj.tles , were computed and t'he distribution 
of\responses to item foils was' examined- A meeting was held with"" the instructor 
Erom 'physical science to explain the role of itera anal^is in item construction. 
Th^ results of ; the analyses along with input from the tutor for the course were 
used in the revision ^f items for that course. Additional meetings with^ the 
other instructors were planned. 

fy) Client satisfact;lpn . The questionnaire solicited responses to items 
about: the management of the Center-, the Center' s*role in the - students' 
educational improvement, the' attitudes ^toward the. tutors, and wh^her or not 
^ Jthe ^tudents like coming to the Center -(see Table 4K / * 



TABLE 4; ATTITUDE QUESTIONNAIRE RESPONSES ' ' * '"1* 

Positive ' "• Neutra.1 _ ^ Regqjrive 
N * 7o • N 7o " N . % 



(1) Management 
function 


•1157 


" 55%^ 


■ ■ 604 


29% 


346 


16% 


•(2)S^ducational 
' betterment 


151 


■ '24% • 


321 


>51%\ 


i » 

159 


25% 


(3) Tutoring . ' 
attitudes 

V 


505 


63%' 


- 213 


27% 


V-78 


■ 10% 


(4) " Like cpiniiig 
. to Teaching 
• Center ' 


i 


« 

•32% 


• . 93 


• 44% 


51 


i 


"General Attitude*' 
(Totals) 


1749- 


547, 


1018 


32% 


457 

— ; A. 


. 14% 



•.Students were strongly favorable, toward the teatTing and gy^ding procedures 

The only areas of * Weakness pinpointed were the privacy of # the grading ar^a and 

the setting of deadline dates ♦ 

Generally no opitiioft was expressed about the extent to which the Center 

contributed toward^ educational hatterment. Some students felt th^t they might 

- ' . ' ^ ' .V 
hav^ trouble adjusting to regular courses, bedause those courses did not 

allow for mi^ftiple testimg and tutoring. , - ^ 

The^ tjLitoring corapojieat "received 'the greatest proportion- of the favorable 

responses* A-majdrity preferred individual tutoring sessions; only twelve 

p^ercent preferred group sessions. - 

* • ' ■ j» 
The" social value of the Center was assessed by^ asking the students> if 

.they liked coming. The. largest: category of responses was neutral. When the- * 

que;5tionnaire was administered*, the Center was- locaJ>eVi in a temporary' buy.ding 

which wgs in poor repair. Subsequently, the Center was m9ved to a more 

attractive location, and a folibw-up questionnaire .showed a substantial shift 

toward' favorable responses on this it^em. . * s * 



SU>IM;\TIVE EVt\LUAJION* 



*l^e Registrar maintains a computer-based data file on students which . 
dates back to 1972. This data -base >rovicJed records for the study. The 
design pxovi'ded' foij^^V longitudinal analysis of t^e success of the five 
grqups of specidl admission students adiaitted from 1970-1975. Criteria for 
success wacs measuted by retenticin, ^gradc point average, passing related non* 

Center courses, and choice of major. The relafe^ionship between' grade point 

m • ' ^ • ^ 

'» 

average and -Florida Tt^elfth, Grade XeSt scores was also examined. 

. - ' ^ llesult^ " • / • , ^ 

A * 

Retention > Retention was, determined by grouping th^ speci^il admission 

students by the year in which they enrolled as freshmen. Retention rates 

increased each year. <For- example, twenty percent of the 1972 students had 
** - ^ \ ' * ^ , 

* graduated T)y 1?76 (se& Table 5). * ' ' ^ ' 

(■-.■■ ' '" '^ ' ■ ^ -: 

TABLE 5. ANALYSIS OF WITHDRAWALS 1972-1976* - ° 
High Achievers and Center Students 



4 


'Graduated 


Withdrew 


Still Enrolled- . 


High achievers 424 ' ^ 


^=110 . - 


■ n'= 163 ■ 
■ ,7o = 38 . 


" N'='151*. 

7o = 36 ' • 

• « 


Center students 118 


N ^ 24 
.>^= 20 


N = • 45 
^ 7. = ' 3 .8 • 


U = • 49. , ■ 
• ' 7o-= 41 .. 



*A':^ur year study ' (1972-19/6) followed a group'of students j^itla ^CJuEP 
credit and a compat"ison group of students with equal ability to^ 

♦ determine -their patterns of a^cademjlc. achievem|^t.. The figures for 
the high schievers iri^able 5 were retention rat^s for the comparison 

/ group in the CLEP study. ' ' ' ^ - ' 



gj^ortyrone percent of -the students were^stilT enrolled', and fchirty-eight^ 
percent had withdrawn. Of those students wha withdrew, -slightly more than- 
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-Iialf were in academic difSiculty. TI^^ priacii^dl dif f erence^betw^ea the* i-pvcial 

adraissLon studentl^s and a group of highSaclTLLej^d^_^£M for which retention 

^ata was available was the reason. for withdrawal. Thir»y-^ix percent of the 

high achieving students who withdres^ were in ticad^niic difficulty compared to 

sixty-two percent of. the special admission stud-ents/who withdrew. More of the 

high achieving students had .gr^duat^d, iut rfB^re of th'e special 'adaiis's ion 

'students were still enrolled. * • . ' - ' . ^ 

Grades . Grades for special admission student^ approximated the average 

grade points fpr all freshmea. Grades earried in Center courses 'were higher 

than those earned for 6t]ier courses, perhaps because of the lujultiple .testing 

' ' * - 

opRgrUtinities offered i;i the Center. 

Do Center courses predict success in related course.s? The;, problea was 

•defined as the relationship between the last course in a subject area taken . 

. . - ' • ^ * - \. " - « ' . • " 

through the Center dnd thje first course^ taken outside, the Center. Passing 

rates for the social sciences were uniformly high regardless of whether or » 

not the students had .ta{^en courses thrjough the Center. ^Passing rates for 

biological science Were uniformly low. In physical, science courses^ passing 

rat'e^s increased f rom^ 1970 tp 1975-fQl: students who took preyio^is courses in 

physical science through ^Ke Center. Similar increases were not fo^^^for^ , 

.students who did not enroll first in: 'Center courSe^^ in 'physical science; 

Major fields .. The grfeat4§t nun\qeif of special admissipn students ^ > 

declared majors', itr sociaJL "science, v/ith education second and* business* third. ' 

Enrollmetit in "education .declined and increase^ in health-rplat^d "fields. 
■* • ' • ^, .■ ' 

, Florida Twelfth G^a^e Test and .grade poirit. average . When all special, 

admipsaon students Were ranked from low to* high on the Florida Twelfth Grade .* 

Test alon^ with their fresjiman and sophomore gr^de point average's, no pUfetern 

'was ajiparent. Achievement t^st scores did not predict the retention of 

students in the special admission category. ' / \ ^ 
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DISSEMINATION 



The evaluation projjuced data related to jnanagement efficiency and pro- • ^ 
gram "accountability. However) so^tte deficiencies dn communication between the 
Center and the students, faculty ^nd' other administrative units were evident. 
-These deficiencies were due not only to » the failure to share, information, but 
' dlsb to the difficulty^ in identifying the apprdpriate offices ^th which to ^ 
sh^re the data. , In a, large university, considerable effort sljouXd be expended 
in locating those persons with whom a communality of interest exists. * v- 

The process of identifying the audience to receive the reports has a 
direct effect upon the type^of information collected and^pon its delivery 
system. Reports tend to vary in purpose,, form, and in the amoujij of detail 
included depending upon the intended audience. The evaluation project data, 
had to be reworked to meet the requirements ^fi^^he Teaching Center staff, the 
federal government, .stude,nts, administrator3^^&fd other universities interested 

in the program. * ^ i^^^^^'^* ^"^'^^K 

When tlye pui^se.of a report was to '-provide data for the development of 
.^^^^^^ * ' " '* ' V. ' - 

efficient management', fo'rmal evaluation mo^s were usedS^and extensive detail 

* . . . ^ ' / " * 

w:as included. However, f etleral* re{>orting recjuirements^ were different! These 
'"reports necessitated the development of particular data in *the form specif ieci 
. by .the Office of fiiealth, kducation, and Welfare.. ^ A thii^ type of report, for . ^ , 
Students, was written. in a question and answer format and was related to problems 
raised by , the students. Finally, reports td^dministrator^ summa'rized accoiont- ^ 
ability* data and made policy xecommendatians. These reports were most eff^c- 
tiv^wrien they were short and concise. . ^ 

f-^y ' . • • . ' ■ / 

$ - , , ; 
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The first tasli undertaken to improve coininiinrcation within 'the Center v/as 

4 

the 'development of a technical manual in which the duties of the staff and the, 

policies and procedures were delineate,d. Even though the manual required 

constant revision, it did provide a record of the' development of the Cent*er 

which was useful in- the evaluation and was ajreference point for .all decision 

making. .The manual, helped to insure that some continuity was preserved during 

the development stage of the'Center. The teclihicai manual later became available 

as a reference for other institutions who were seeking information on the opera- 

-Jtijon of programs for^peci^l admission students. 

,^ A major obstacle to coiranunication within the University was the scarcity 

of available' data on the "target population o'f students. Even with a. computer- 

.based information system at the University, it was difficult and expensive 

retrieve information on the success of a particular group of students. The 

duplication of effort and the int^ccessibility of data at times resulted in con- 
, ' » *' * 

flicting reports generated on the same students. The -creation of a data base" 

^- . 

for the Teaching Center evaluation provided the opportunity to meet the needs " 
of other university agencies. Meetings were held with all -interested administra^ 
tive units in order to insure that the new data*^ base would meet their reporting 
requirements. The .coordination made possible by the maintenance of a connAon data 
base made communication between units a natural process because it was mutually 
beneficial. ' t ' . * • . 

Procedural problems with the faculty and students were often directly 



related tCLP^o^ communication:. Therefore^ a faculty newsletter was distributed 

*queu:terly» The newsletter gave feedback from the evaluation 9a the effects on 

grades due to repeat testing, and any procedural problems which had emerged. * 

>. ^ " * • 

In addition, information about related programs in reading and writing skills 



, development or other -services was transmitted^. Weekly reports on stU(2ient prog- 
ress v;ere also , sent/ to faculty, the athletic department and DSSSP. 

In o'rder to keep students well informed, posters and graphs wer^e dis- • 
play^(i in the- Center. For example, a ^ow chart w4s color, keyed to represent ^ 
test deadlines in course sections. In this way', students were encouraged- to 
avoid testing during pe^ use periods. Leaflets were also prepared to respond 
to questions or problems students raised on the c|.ient satisfaction que^tiion-_ 
naire* Feedback to &tudenibs ftom client satisfaction data was , expected io 

\^ improve the cjuality of tite jcesponses to future questionnaires as well as provide 
an evaluation, of student attitudes toward the Center, 



Finally, quarterly progress reports have been distributed within 



University which- distail accountability information. The reports include data 



x^tic 



the 



, on the number axSl type of course sections using the Center services, the number 

of students enrolled in the Center sections, and the academic progresf of the 
• * 

students. Accurate .information dis^seminated thro\ighout the University has been 
in^luable in counteracting suppositions incorrectly h^ld about special students* 

SURMARY AND CONCLUSIONS ^ . ' 

The f^rtn^iye phase of the evaluatj.on of the Teaching (fenter was designed 
to provide information to be used* ia the development of policies. The data in- 
dicated that streamlining the' testing and grading procedures would alleviate 
some qf*the congestion at peak use periods. In p4rticular, limits on the amount 

t 

and frequency of repeat testing'wer^ warranted. ' / 

The overall effectiveness of the tutoring program was established. How- 
ever, the tutoring pr'ograin was not "uniformly siirong in all courses. A tutor 
training program was recommended in order to provide a more" uniform Iquality of 

15--- . . . . . , 



instruction. Finally, the analyses of the item banks did reveal a need for ^ 

further improvement of the quality of the' items. ^ ^ » 

« . 

* -The suipmative evaluation indicated that Center students 'did academically 

survive at the University, and this success was not related to achievement • te'^t . 

^scores. * . - . . . ' ' ^ . . 

The* ultimate success of the Center" depends upon the satisfaction of its 

clients, the students and the faculty. The on-rgoing evaluation effort prbmotes 

< .• . * < * 

a continuous assessment of the Center's activities. The interchange of infonna- 
tion resulting from the evaluation raises questions about policies, procedures, * 
and even the evaluation itself, ^he assumption upon whlth the staff oper^es 



is that evaluation ind dissem^^na^zipn ]^r,omote the questioning which" is th^ key 
to the development *pf a pro^gram satisfying to its clients. 



